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Abstract: In 1996 the Department of Computer Science, Monash University, implemented a First 
Year Advanced Students’ Project Scheme aimed at extending and stimulating its best first year 
students. The aim of the scheme was to give students the opportunity to work on a project that best 
suited their needs and captured their interests. 

One of the projects which became known as CADAL Quiz, (Computer Aided Dynamic Assessment 
& Learning Quiz), involved designing and implementing a World Wide Web (WWW) based 
multiple choice quiz generator and assessment tool. 

Unexpectedly, at the time, there were several academics wishing to move away from the traditional 
mode of educational assessment and move towards interactive, computerised assessment. As a 
result, CADAL Quiz was incorporated into the First Year Computer Programming unit and utilised 
by lecturers, tutors and students. 

This paper reports on a pilot project for developing and integrating CADAL Quiz into the 
curriculum. It highlights the unique quiz features, and its use by students and staff. The paper 
describes how the quiz was incorporated into the First Year Computer Programming unit and 
presents a conduit of attitudes useful to those who are planning to use the Web as a resource for 
educational assessment. 



1. Introduction 

With the onset of the Internet, in particular the World Wide Web (WWW), it has been increasingly 
popular to move away from the traditional mode of education and move towards a more interactive, 
computerised system. [Godfrey, 1996], [Conway, 1993], [Conway, 1994]. Such a move was simplified 
by the work of a devoted student who designed and implemented CADAL Quiz as part of the 1996 
Computer Science, First Year Advanced Students’ Project Scheme [Carbone, 1996], [Carbone, 1996]. 

CADAL Quiz is a multiple choice quiz generator and assessment tool that utilises the WWW. In 1997 
CADAL Quiz was incorporated into the curriculum by first year lecturers. This change from paper based 
assessment to WWW based assessment is described in this paper as well as the various features of the 
CADAL Quiz package. 



2. Design and Description of CADAL Quiz 

CADAL Quiz, as used by the First Year Computer Programming unit, was designed with a number of 
goals in mind, including: 
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1. Providing shared ownership of assessment questions by tutors and lecturers. In the past 
lecturers took the sole responsibility of setting assessment tasks. CADAL Quiz was 
introduced to provide a structure for tutors and lecturers to work as a team and share in the 
responsibility of developing assessment questions. 

2. To encourage metacognitive learning in students. One way of making students more aware of 
their learning is to provide them with self assessment questions that are tied to each week’s 
laboratory tasks enabling them to monitor their own understanding. Unlike ordinary paper 
quizzes, both the student and staff can gain immediate feedback on their understanding and 
results. If a student is unsure of a response or result, they can discuss it with relevant staff 
immediately, rather than waiting for a paper quiz to be marked and returned, when the query 
may be less relevant. 

3. Reducing the opportunity to copy or cheat in tests. As each quiz is unique, it makes it 
difficult for students to cheat. 

4. Cutting the cost of assessment. CADAL Quiz automatically generates and corrects quizzes. 
This reduces staffhours required for printing, administering and marking. 

5. Recording student results. A complete log is kept on who attempted the quiz, when they 
attempted it, specific question choices, time taken and final result. This not only serves as an 
assessment tool, but can be utilised in future curriculum development. 

6. Flexibility for staff to govern the test. Staff decide on the number of questions per quiz, 
specify time slots and passwords to restrict quiz access, have the ability to view quiz logs and 
graph statistics. 

While there are tools to develop interactive lessons on the web, such as SAMaker [Sloane and Dyreson 
1996], and others [John Tasker, 1997], [Indiana University, 1997] many do not generate random sets of 
questions and do not record student results. The quizzes that are also online Eire either rigid or hard 
coded. These types of quizzes have limited applications and are generally useful for student self- 
assessment only. In overcoming the restrictions of conventional paper quizzes, a number of features were 
incorporated into CADAL Quiz, including: 



2.1 Random ordering of questions and the A, B, C, D choices. 

Quizzes are generated from a database of questions. This involves randomly selecting a specified number 
of questions from the database, and randomly ordering the chosen questions. The multiple choice 
alternatives (A, B, C, D) are also randomly ordered, so that if two students have the same question, the 
questions will appear to be different, which minimises cheating. 



2.2 Results handling and analysis features 
a) Immediate assessment with logged details. 

The fact that each quiz is corrected immediately is a major advantage over paper quizzes. At the same 
time, student results are logged and can be analysed immediately. Logged information includes the 
student's name, ID number, email address, demonstrator's email address, the quiz attempted, the time it 
was attempted, the time taken to complete the quiz, the student’s response to each question and the final 
result. 



b) Results optionally displayed to the student. 

During self-assessment, it is acceptable for the student to see their results immediately. However, in the 
case of a test, results can be hidden from the student until after all students have attempted the test, which 
again aims to minimise cheating. The same applies to emailing students their results for future reference. 
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c) Results optionally emailed to staff/supervisors. 

In the case when the quiz is used as part of laboratory preparation, it is convenient to have the student's 
results emailed to the lab demonstrator for recording. This also applies if the quiz is used as a survey. 
This can be turned off if not required. 

d) Resuits and statistics can be viewed and graphed oniine. 

Logged information can all be viewed online. In the case of student responses to questions and final 
results, these can be graphed online, to indicate the more difficult/easier questions. This helps locate 
student strengths and weaknesses. The fact that this can all be done straight after or during the 
administration of a quiz, means that a difficult topic can be revised as soon as it becomes a problem. 



2.3 The ability to subdue the randomness and specify a question breakdown. 

To ensure that certain questions or topic areas are included in the quiz a question breakdown can be 
specified. Questions can be chosen fi-om a range of questions, for example, select 3 questions from 
questions 1 to 10 or include question 24. This then guarantees that ranges of questions are selected, 
perhaps ensuring that several harder questions are included. 



2.4 Administration features (such as adding and viewing questions online). 

To make it easier for staff to insert questions into the quiz databases, questions can be added online. This 
includes password restricted access and step by step instructions to adding a new question. Staff can also 
view all questions in the database online, as opposed to a student who only sees a random portion. 



2.5 Restricted access to quizzes and secure staff areas. 

Most quizzes are available at any time, but in the case of tests it might be necessary to restrict access to 
certain people at various times. For example a test can be conducted over the course of a week, and only 
certain people can access it at any one time using time specific passwords. Access to staff areas is also 
password restricted. 



3. Integrating CADAL Quiz into First Year Computer Programming 

In the past it has been common practice to assess First Year Computer Science students through 
laboratory exercises, a multiple choice mid-semester test and an exam. The laboratory exercises are 
marked out of 10; 3 marks towards preparation and 7 marks devoted to the programming exercises. The 
mid-semester test was conducted during the lecture and counts towards 10% of the student's overall result 
for the subject [Farr and Nicholson, 1996]. 

With the traditional practices of assessment there has been concern about students copying preparation 
work and whether the mid semester test was cost-effective given the current pressure on resources and 
budget cuts. This year, CADAL Quiz changed the way in which students were assessed. 

Each week a number of tutors devised and submitted a set of multiple choice questions into the database. 
These questions were related to the current week’s laboratory task and aimed at testing whether the 
students had adequately prepared for their laboratory tasks and understood the abstractions of the lesson. 

Students generated and attempted CADAL Quizzes during three practical classes. These quizzes 
contained 10 questions, chosen from a much larger set, and contributed to the preparation component of 
the student’s practical mark for that week. In general the quiz took on average, approximately 10-15 



minutes in most classes to administer and complete and in that time the students and tutors received 
details of the student’s attempt. Some of the details shown in Table 1 below included: the date when the 
quiz was taken, the questions answered and a score out of 10. These results were automatically mailed to 
the demonstrator and counted towards 3 preparation marks. 



Date: 


Mon, 14 Apr 1997 14:09:35 +1000 (EST) 


From: 


Online CSClOl 1 Quiz 


Subject: 


Student Results - CSClOl 1 Quiz 


Reply-To: 


usemame@student.monash.edu.au 


Supposedly-From 


: John Smith 


Student: 


John Smith 


ID: 


12345678 


Demonstrator: 


tutor@cs.monash.edu.au Quiz: csclab3 


Date & Time: 


Mon Apr 14 14:09:34 1997 


Results: (3=B) (1 1 


=C) 4 (17=B) 21 (29) 1 (18=A) 24 2 


** TOTAL: 5 out of 10** 


Key: X is correct, (X=Y) is incorrect, (X) is not answered. 
(Where X is the question number and Y their incorrect choice.) 



Table 1 : Sample email sent to staff and students 



During week 5 the students were familiar with the operations of CADAL Quiz so the lecturers used 
CADAL to replace the traditional mid-semester test, (which was formerly run in the lecture theatre over 
two lectures). The mid-semester test contained 50 questions chosen from approximately 165 questions and 
was held in the computer labs over a period of one week. The randomness of the questions was subdued 
so that tests of comparable difficulty were generated. 



4. Responses - Statistics and the Educational Impact 

During its first semester of operation, the impact of incorporating CADAL Quiz was measured by 
gathering student performance statistics and perceptions from tutors and lecturers. From the first trial 
there have been beneficial effects for the tutors and course lecturers as well as students. 



4. 1 Tutor and Demonstrator Responses 

Teaching staff were surveyed to provide feedback on the effect CADAL Quiz had on the operations of the 
laboratory classes. They were also interviewed to discuss the feasibility of designing and shaping 
educational assessment tasks in groups, with combined tutor and lecturer involvement. 

Although the process of formulating questions and adding them to the general pool via the Web increased 
the sense of shared-ownership felt by the tutors there were several deficiencies in structure and execution 
of the quiz identified by the group. These included: 

errors in the wording of submitted question, 
students could make multiple submissions, 

the random ordering of questions did not ensure that all quizzes were of an equal level of difficulty, 
difficulty in helping students when they answered a question incorrectly because it was difficult to tell 
which question they answered due to the random ordering 

As a result of the above observations, the design of CADAL Quiz was changed prior to the mid semester 
test to provide focused, more personalised assistance to the students. The significant changes were: 
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test questions were attempted and proof read by three independent tutors for better monitoring, 
students were only allowed to make one submission, 

addition of the ability to subdue the random generation of questions to produce tests of equal difficulty, 
releasing the total database of questions and answers, after the test was completed by all students, so 
that students could tell which question they answered, and the option they selected 

The majority of tutors (70%) believed that the quiz was an effective way of determining whether a student 
had adequately prepared for the lab. Errors, both system and question design were rarely encountered. 



4.2 Lecturer Review 

The above changes in the structure and execution of the CADAL Quiz appear to have been very 
successful. Indeed feedback from lecturers was very positive under the revised fi^mework. 

CADAL Quiz automatically compiles and graphically displays the alternatives students selected for each 
quiz. The lecturers found this information very interesting and the online graph of the overall 
performance on each question very appealing. Not only has this enabled easy detection of the hardest 
questions (ie. most wrong answers) but even which wrong answer is most commonly selected. As a result, 
lecturers have received hitherto unknowable feedback about the meanings their students are constructing. 

"It really did help me pick up quickly on where the strong and weak points are.... It was very good to be 
able to see, at a glance, which questions they were very good at, which questions they were on average 
completely clueless about (4 bars of roughly similar length), which questions they had some vague idea 
about but were thwarted when it comes to detail (perhaps a couple of good sized bars, other small bars), 
and which (few) questions completely threw them " Graham Farr, CSC 1011 Lecturer 

The graph produced from student results determined CADAL Quiz’s usefulness in steering course design. 
Difficult and easy questions were highlighted so that course lecturers could accurately locate the most 
misunderstood topics, or poorly worded questions and answers. This has allowed improvements to 
teaching while the course is still running. 

"I will be looking in future lectures to further emphasise some of the many points where they are weak.. ” 
Graham Farr, CSC 1011 Lecturer 



4.3 Student Results 

CADAL Quiz was particularly useful in making students more aware of their own learning. In particular 
students decided whether they needed to do one or more of the randomly generated quizzes. These 
thought processes are all associated with enhanced metacognition. 

With respect to the mid-semester test, a total of 365 students completed the test of 50 questions, with an 
average result of 60% (Standard Deviation 7.55, Range 7 - 49), which is comparable to the 1996 mid- 
semester (traditional hardcopy test) result, where 395 students sat the test and received an average mark of 
60% (Standard Deviation 7.53, Range 7 - 49). With CADAL Quiz there was no indication that the 
students who completed the test later in the week had an advantage. The percentage of correct answers 
for each question varied from 98.5% (1 incorrect response out of 67) indicating a particularly easy 
question, right down to 10% correct and below, indicating particularly hard or poorly worded questions. 




6 



5. Conclusion and Future directions 



CADAL Quiz is an application of WWW technology that has had a significant impact on educational 
assessment materials on the Internet. The process of formulating questions has increased the sense of 
shared ownership felt by tutors for the course. The lecturers have received hitherto unknowable feedback 
(from the aggregate statistics) about student's understanding and the meanings their students are 
constructing. This has allowed lecturers to adjust their teaching while the course is still running. 

The quizzes were particularly useful in testing the students understanding. The personalised feedback 
made the students more aware of their own learning, hence enhancing their metacognitive skills. 

CADAL Quiz has several advantages over paper based quizzes. These include ease of automatic marking, 
ease of creation of individualised tests, and immediate feedback to students. Continued changes and 
improvements will make CADAL Quiz one of the most functional Web-based testing methods available. 
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